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Abstract 

Let £ be a class of probability distributions over the discrete domain [n] = {1, . . . ,n}. We 
show that if £ satisfies a rather general condition - essentially, that each distribution in £ can be 
well-approximated by a variable-width histogram with few bins - then there is a highly efficient 
(both in terms of running time and sample complexity) algorithm that can learn any mixture 
of k unknown distributions from £. 

We analyze several natural types of distributions over [n] , including log-concave, monotone 
hazard rate and unimodal distributions, and show that they have the required structural prop- 
erty of being well-approximated by a histogram with few bins. Applying our general algorithm, 
we obtain near-optimally efficient algorithms for all these mixture learning problems as described 
below. More precisely, 

• Log-concave distributions: We learn any mixture of k log-concave distributions over 
[n] using k ■ 0{l/e^) samples (independent of n) and running in time 0(fc log(7i)/e^) bit- 
operations (note that reading a single sample from [n] takes 8(logn) bit operations). For 
the special case fc = 1 we give an efficient algorithm using 0(1/ e^) samples; this generalizes 
the main result of |DDS12b] from the class of Poisson Binomial distributions to the much 
broader class of all log-concave distributions. Our upper bounds are not far from optimal 
since any algorithm for this learning problem requires f2(fc/e^/^) samples. 

• Monotone hazard rate (MHR) distributions: We learn any mixture of k MHR 
distributions over [n] using 0{k\og{n/e)/e^) samples and running in time 0(fc log^(n)/e'^) 
bit-operations. Any algorithm for this learning problem must use fl{k\og{n)/e^) samples. 

• Unimodal distributions: We give an algorithm that learns any mixture of k unimodal 
distributions over [n] using 0(fc log(n)/e^) samples and running in time 0(fc log^(n)/e'^) 
bit-operations. Any algorithm for this problem must use fl{klog{n)/e^) samples. 
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1 Introduction 



1.1 Background and motivation. Learning an unknown probability distribution given access 
to independent samples is a classical topic with a long history in statistics and probability theory. 
Theoretical computer science researchers have also been interested in these problems at least since 
the 1990s [KMR"'"94 IDas99] . with an explicit focus on the computational efficiency of algorithms 



for learning distributions. Many works in theoretical computer science over the past decade have 
focused on learning and testing various kinds of probability distributions over high-dimensional 
spaces, see e.g. |Das99l [FM991 iPgOOl IXKOTl I V W021 [FUHOSl [R505l [BgTOl IKMVlOl IMVlOl lACSlOj 
and references therein. There has also been significant recent interest in learning and testing various 
types of probability distributions over the discrete domain [n] = {1, . . . , n}, see e.g. jBKR04tlVVllbt 
IVVllal IDDS12al IDDS12b| . 

A natural type of distribution learning problem, which is the focus of this work, is that of learning an 
unknown mixture of "simple" distributions. Mixtures of distributions have received much attention 
in statistics |Lin95l IRW841 ITSM85j and in recent years have been intensively studied in computer 
science as well (see many of the papers referenced above). Given distributions pi, ■ ■ ■ ,Pk and non- 
negative values /ii, . . . , /ifc that sum to 1, we say that p = X^JL]^ fiiPi is a k-mixture of components 
pi, . . . ,Pk with mixing weights /xi, . . . ,/Ufc. A draw from p is obtained by choosing i € [k] with 
probability and then making a draw from pi. 

In this paper we work in essentially the classical "density estimation" framework |Sil861 ISco921 
IDLOlj which is very similar to the model considered in |KMR"'"94| in a theoretical computer science 
context. In this framework the learning algorithm is given access to independent samples drawn 
from an unknown target distribution over [n] , and it must output a hypothesis distribution h over 
[n] such that with high probability the total variation distance d^vip^h) between p and h is at 
most £. Thus, for learning mixture distributions, our goal is simply to construct a high-accuracy 
hypothesis distribution which is very close to the mixture distribution that generated the data. In 
keeping with the spirit of [KMR"'"94] . we shall be centrally concerned with the running time as 
well as the number of samples required by our algorithms that learn mixtures of various types of 
discrete distributions over [n]. 

We focus on density estimation rather than, say, clustering or parameter estimation, for several rea- 
sons. First, clustering samples according to which component in the mixture each sample came from 
is often an impossible task unless restrictive separation assumptions are made on the components; 
we prefer not to make such assumptions. Second, the classes of distributions that we are chiefly 
interested in (such as log-concave, MHR and unimodal distributions) are all non-parametric classes, 
so it is unclear what "parameter estimation" would even mean for these classes. Finally, even in 
highly restricted special cases, parameter estimation provably requires sample complexity exponen- 
tial in k, the number of components in the mixture. Moitra and Valiant [MVlOj have shown that 
parameter estimation for a mixture of k Gaussians inherently requires exp(r2(A:)) samples. Their 
argument can be translated to the discrete setting, with translated Binomial distributions in place 
of Gaussians, to provide a similar lower bound for parameter estimation of translated Binomial 
mixtures. Thus, parameter estimation even for a mixture of k translated Binomial distributions 
over [n] (a highly restricted special case of all the mixture classes we consider, since translated Bi- 
nomial distributions are log-concave, MHR and unimodal) requires exp(r2(/c)) samples. This rather 
discouraging lower bound motivates the study of other variants of the problem of learning mixture 
distributions. 

Returning to our density estimation framework, it is not hard to show that from an information- 
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theoretic perspective, learning a mixture of distributions from a class <t of distributions is never 
much harder than learning a single distribution from C In Appendix [X] we give a simple argument 
which establishes the following: 

Proposition 1.1. [Sample Complexity of Learning Mixtures] Let ^ be a class of distributions over 
[n]. Let A be an algorithm which learns any unknown distribution p in ^ using m{n,e) ■ log(l/5) 
samples, i.e., with probability 1 — 5 A outputs a hypothesis distribution h such that dTvip^h) < 
e. where p ^ <t is the unknown target distribution. Then there is an algorithm A' which uses 
0{k/e^) •m(n, e/20) •log^(l/(5) samples and learns any unknown k-mixture of distributions in C to 
variation distance e with confidence probability 1 — 6. 

While the generic algorithm A' uses relatively few samples, it is computationally highly inefficient, 
with running time exponentially higher than the runtime of algorithm A (since A' tries all possible 
partitions of its input sample into k separate subsamples). Indeed, naive approaches to learning 
mixture distributions run into a "credit assignment" problem of determining which component 
distribution each sample point belongs to. 

As the main contributions of this paper, we (i) give a general algorithm which efficiently learns 
mixture distributions over [n] provided that the component distributions satisfy a mild condition; 
and (ii) show that this algorithm can be used to obtain highly efficient algorithms for natural 
mixture learning problems. 

1.2 A general algorithm. The mild condition which we require of the component distributions 
in our mixtures is essentially that each component distribution must be close to a (variable- width) 
histogram with few bins. More precisely, let us say that a distribution q over [n] is {£,t)-fiat (see 
Section [2|) if there is a partition of [n] into t disjoint intervals such that p is e-close 
(in total variation distance) to the distribution obtained by "flattening" p within each interval Lj 
(i.e., by replacing p{k), for k E Lj, with X^jg/ .p(i)/l-^jl)- general result for learning mixture 
distributions is a highly efficient algorithm that learns any /c-mixture of (e, t)-flat distributions: 

Theorem 1.1 (informal statement). There is an algorithm that learns any k-mixture of [e,t)-flat 
distributions over [n\ to accuracy 0{e), using 0{kt/e^) samples and running in 0{ktlog(n)/e^) 
bit- operations. 

As we show in Section 11.31 below. Theorem 11.11 yields near-optimal sample complexity for a range 
of interesting mixture learning problems, with a running time that is nearly linear in the sample 
size. Another attractive feature of Theorem 11.11 is that it always outputs hypothesis distributions 
with a very simple structure (enabling a succinct representation) , namely histograms with at most 
kt/e bins. 

1.3 Applications of the general approach. We apply our general approach to obtain a wide 
range of learning results for mixtures of various natural and well-studied types of discrete distri- 
butions. These include mixtures of log-concave distributions, mixtures of monotone hazard rate 
(MHR) distributions, and mixtures of unimodal distributions. To do this, in each case we need 
a structural result stating that any distribution of the relevant type can be well-approximated by 
a histogram with few bins. In some cases (unimodal distributions) the necessary structural re- 
sults were previously known, but in others (log-concave and MHR distributions) we establish novel 
structural results that, combined with our general approach, yield nearly optimal algorithms. 
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Log-concave distributions. Discrete log-concave distributions are essentially those distributions 
p that satisfy > p{k + l)p{k — 1) (see Section U] for a precise definition) . They are closely anal- 

ogous to log-concave distributions over continuous domains, and encompass a range of interesting 
and well-studied types of discrete distributions, including binomial, negative binomial, geometric, 
hypergeometric, Poisson, Poisson Binomial, hyper-Poisson, Polya-Eggenberger, and Skellam dis- 
tributions (see Section 1 of |FBRllj ). In the continuous setting, log-concave distributions include 
uniform, normal, exponential, logistic, extreme value, Laplace, Weibull, Gamma, Chi and Chi- 
Squared and Beta distributions, see |BB05j . Log-concave distributions over [n] have been studied 
in a range of different contexts including economics, statistics and probability theory, and algebra, 
combinatorics and geometry, see |An951 IFBRIH ISta89j and references therein. 

Our main learning result for mixtures of discrete log-concave distributions is: 

Theorem 1.2. There is an algorithm that learns any k-mixture of log-concave distributions over 
[n] to variation distance e using k ■ 0(l/e^) samples and running in 0{k\og{n) / e'^) hit- operations. 

We stress that the sample complexity above is completely independent of the domain size n. In 
the special case of learning a single discrete log-concave distribution we achieve an improved sam- 
ple complexity of 0(l/e^) samples, with running time 0(log(n)/e^). This matches the sample 
complexity and running time of the main result of |DDS12b] . which was a specialized algorithm 
for learning Poisson Binomial distributions over [n]. Our new algorithm is simpler, applies to the 
broader class of all log-concave distributions, has a much simpler and more self-contained analysis, 
and generalizes to mixtures of k distributions (at the cost of an additional 1 /e factor in runtime and 
sample complexity). We note that these algorithmic results are not far from the best possible for 
mixtures of log-concave distributions. We show in Section |4] that for k < n^~^(^^ and e > l/n^^^\ 
any algorithm for learning a mixture of k log-concave distributions to accuracy e must use Q{k/e'^'^) 
samples. 

Monotone Hazard Rate (MHR) distributions. A discrete distribution p over [n] is said to 
have a monotone {increasing) hazard rate if the hazard rate H(i) *== ^ ^^^^^ — is a non-decreasing 

function of i. It is well known that every discrete log-concave distribution is MHR (see e.g. part 
(ii) of Proposition 10 of |An95j ). but MHR is a more general condition than log-concavity (for 
example, it is easy to check that every non-decreasing distribution over [n] is MHR, but such 
distributions need not be log-concave). The MHR property is a standard assumption in economics, 
in particular auction theory and mechanism design |Mye81 , IFT9H IMCWG95] . Such distributions 



also arise frequently in reliability theory; |BMP63j is a good reference for basic properties of these 
distributions. 

Our main learning result for mixtures of MHR distributions is: 

Theorem 1.3. There is an algorithm that learns any k-mixture of MHR distributions over [n] to 
variation distance e using 0{k\og{n / e) / e'^) samples and running in 0{k\o^ {n) / e'^) hit- operations. 

This theorem is also nearly optimal. We show that for k < n^~^^^^ and e > l/n^^^\ any algorithm 
for learning a mixture of k MHR distributions to accuracy e must use Q{k\og{n)/e^) samples. 

Unimodal distributions. A distribution over [n] is said to be unimodal if its probability mass 
function is monotone non-decreasing over [l,t] for some t < n and then monotone non-increasing 
on [t,n]. Every log-concave distribution is unimodal, but the MHR and unimodal conditions are 
easily seen to be incomparable. Many natural types of distributions are unimodal and there has 
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been extensive work on density estimation for unimodal distributions and related questions |Rao69t 
Weg70t IBKR04[ [Bi?97l ITbu97] • 



Our main learning result for mixtures of unimodal distributions is: 

Theorem 1.4. There is an algorithm that learns any k-mixture of unimodal distributions over [n] 
to variation distance e using 0{klog{n)/e'^) samples and running in 0(A;log^(n)/e^) bit- operations. 

Our approach in fact extends to learning a fc-mixture of t- modal distributions (see Section [6]). The 
same lower bound argument that we use for mixtures of MHR distributions also gives us that for 
k < n^~^^^') and e > l/n^^^\ any algorithm for learning a mixture of k unimodal distributions to 
accuracy e must use ^{klog{n)/e'^) samples. 



1.4 Related work. Log-concave distributions: Maximum likelihood estimators for both con- 
tinuous |DR091 IWal09j and discrete [FBRllj log-concave distributions have been recently studied 
by various authors. For special cases of log-concave densities over M (that satisfy various restric- 
tions on the shape of the pdf) upper bounds on the minimax risk of estimators are known, see 
e.g. Exercise 15.21 of |DL01j . (We remark that these results do not imply the k = 1 case of our 
log-concave mixture learning result.) Perhaps the most relevant prior work is the recent algorithm 
of |DDS12b] which gives a 0(l/e^)-sample, 0(log(n)/e^)-time algorithm for learning any Poisson 
Binomial Distribution over [n]. (As noted above, we match the performance of the |DDS12b] al- 
gorithm for the broader class of all log-concave distributions, as the k = 1 case of our log-concave 
mixture learning result.) 

Achlioptas and McSherry |AM05j and Kannan et al. |KSV08j gave algorithms for clustering points 
drawn from a mixture of k high-dimensional log-concave distributions, under various separation 
assumptions on the distance between the means of the components. We are not aware of prior work 
on density estimation of mixtures of arbitrary log-concave distributions in either the continuous or 
the discrete setting. 

MHR distributions: As noted above, MHR distributions appear frequently and play an impor- 
tant role in reliability theory and in economics (to the extent that the MHR condition is considered 
a standard assumption in these settings). Surprisingly, the problem of learning an unknown MHR 
distribution or mixture of such distributions has not been explicitly considered in the statistics 
literature. We note that several authors have considered the problem of estimating the hazard rate 
of an MHR distribution in different contexts, see e.g. |Wan861 [HW93| \G.ni\ fBannSj . 

Unimodal distributions: The problem of learning a single unimodal distribution is well-understood: 
Birge |Bir97j gave an efficient algorithm for learning continuous unimodal distributions (whose 
density is absolutely bounded); his algorithm, when translated to the discrete domain [n], requires 
0(log(n)/e'^) samples. This sample size is also known to be optimal (up to constant factors) [Bir87aj . 
In recent work, Daskalakis et al. |DDS12a] gave an efficient algorithm to learn t-modal distribu- 
tions over [n]. We remark that their result does not imply ours, as even a mixture of two unimodal 
distributions over [n] may have J7(n) modes. We are not aware of prior work on efficiently learning 
mixtures of unimodal distributions. 

Paper Structure. Following some preliminaries in Section [21 Section [3] presents our general 
framework for learning mixtures. Sections [H [5] and [6] analyze the cases of log-concave, MHR and 
unimodal mixtures respectively. 
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2 Preliminaries and notation 



We write [n] to denote the discrete domain {1, . . . , n} and to denote the set {i, ■ ■ ■ , j} for i < j. 
For V = (f(l), • • • ,v{n)) G M" we write ||u||i = l^(*)l to denote its Li-norm. 

For p a probabihty distribution over [n] we write p{i) to denote the probabihty of element i G [n] 
under p, so p{i) > for all i G [n] and = 1- For S C we write p{S) to denote 

We write p'^ to denote the sub- distribution over S induced by p, i.e., p'^(i) = p{i) if 
i G S" and p'^(z) = otherwise. 

A distribution p over [n] is non-increasing (resp. non-decreasing) if p{i -|- 1) < p{i) (resp. p{i -|- 1) > 
p{i)), for all i G [n — 1]; p is monotone if it is either non-increasing or non-decreasing. 

Let p,q be distributions over [n]. The total variation distance between p and q is d^viP^Q) =^ 
max5c[n] \p{S) — q{S)\ = (1/2) • \\p — q\\i. The Kolmogorov distance between p and q is defined as 

dKip,q) =^ maxjg[„] X^-=iP(i) - Ei=i • Note that dK{p,q) < dTv{p,q)- 



Finally, the following notation and terminology will be useful: given m independent samples 
si, . . . , Sm, drawn from distribution p : [n] [0,1], the empirical distribution pm ■ [n] [0,1] 
is defined as follows: for all i € [n], Pm{i) = |{i G [m] \ sj = i} \ /m. 

Partitions, flat decompositions and reflnements. Given a partition I = {Ji,...,/t} of 

[n] into t disjoint intervals and a distribution p over [n], we write p^^^(^) to denote the flattened 
distribution. This is the distribution over [n] defined as follows: for j G [t] and i Ij, p^^^^-^\i) = 
p{Ij)/\Ij\. That is, pfl^t(^) is obtained from p by averaging the weight that p assigns to each interval 
in Z over the entire interval. 

Deflnition 2.1 (Flat decomposition). Let p be a distribution over [n] and V be a partition of [n] 
into t disjoint intervals. We say that V is a (p, e,t)-fiat decomposition of [n] if dTviP^P^^^^^^) ^ ^• 
If there exists a {p,e,t)-flat decomposition of [n] then we say that p is (e,t)-fiat. 

Let X = {Ii, . . . , Is} be a partition of [n] into s disjoint intervals, and J = {Ji, . . . , Jt} be a 
partition of [n] into t disjoint intervals. We say that J7 is a refinement of I if each interval in I is 
a union of intervals in J', i.e., for every a G [s] there is a subset Sa ^ [t] such that la = Ub^SaJb- 

For I = {Ii}l^i and I' = {II}^^^ two partitions of [n] into r and s intervals respectively, we say 
that the common refinement of I and Z' is the partition J of [n] into intervals obtained from Z 
and Z' in the obvious way, by taking all possible nonempty intervals of the form Ij n /j. It is clear 
that J' is both a refinement of Z and of Z' and that J' contains at most r + s intervals. 



2.1 Basic Tools. We recall some basic tools from probability. 

The VC inequality. Given a family of subsets A over [n], define ||p||_4 = supyig_4 |p(^)|. The 
VC-dimension of A is the maximum size of a subset X C [n] that is shattered by A (a set X is 
shattered by A if for every Y C. X some A G ^ satisfies ^ H X = 1"). 

Theorem 2.1 (VC inequality, |DLni[ p.31]). Let Pm be an empirical distribution of m samples 
from p. Let A be a family of subsets of VC-dimension d. Then 

E[\\p-pm\\A]<0{Vd/^). 



Uniform convergence. We will also use the following uniform convergence bound: 
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Theorem 2.2 ( |DLOH pl7]). Let A be a family of subsets over [n], and pm be an empirical 
distribution of m samples from p. Let X be the random variable \\p — p\\j^^- Then we have 

Fj:[X-E[X] >7]] <e-2'"'''. 
3 Learning mixtures of (e, t)-flat distributions 

In this section we present and analyze our general algorithm for learning mixtures of (e, t)-flat 
distributions. We proceed in stages by considering three increasingly demanding learning scenarios, 
each of which builds on the previous one. 

3.1 First scenario: known flat decomposition. We start with the simplest scenario, in which 
the learning algorithm is given a partition V which is a {p, e, t)-flat decomposition of [n] for the 
target distribution p being learned. 

Algorithm Learn-Known-Decomposition(p, V, e, 5): 

Input: sample access to unknown distribution p over [n]; (p, e, i)-flat decomposition V of [n]; 
accuracy parameter e; confidence parameter 6 

1. Draw m = 0{{t + log samples to obtain an empirical distribution pm- 

2. Return {pmf^^^'^\ 



Theorem 3.1. Let p be any unknown target distribution over [n] andV he any {p,e,t)-flat decom- 
position of [n]. Algorithm Learn-Known-Decomposition(p, 7^, e, (5) draws 0((t + log(l/(5))/e^) 
samples from p and with probability at least 1 — 6, outputs (pm)^'^*''^^ such that d^v {{'Pm)^^^^^\ p) ^ 
2e. Its running time is 0{{t + log(l/(5)) • log(n)/e^) bit operations. 

Proof. An application of the triangle inequality yields 

dTV {p, iPmr'^''^) < dry + dry (/^^(^) , (p^)*!^*^^)) . 

The first term on the right-hand side is at most e by the definition of a (p, e, t)-fiat decomposition. 
The second term is also at most e, as follows by Proposition 13. H stated and proved below. □ 

Proposition 3.1. Let p he any distribution over [n] and let pm be an empirical distribution of 
m = 0((s + logl/(5)/e^) samples from p. Let V be any partition of [n] into at most s intervals. 
Then with probability at least 1-5, dTv(p^''*''^\ (Fm '^^'') < £■ 

Proof. By definition we have 

= (1/2) E W) -pM\ = \\p-PmU 

where A = \_}{I S V \ p{I) > pm{I)}- Since V contains at most s intervals, ^ is a union of at most 
s intervals. Consequently the above right-hand side is at most \\p — Pm\\_A^, where As is the family 
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of all unions of at most s intervals over [n]\}] Since the VC-dimension of As is 2s, Theorem 12.11 
implies that the considered quantity has expected value at most e. The claimed result now follows 
by applying Theorem 12.21 with rj = e. □ 



3.2 Second scenario: unknown flat distribution. The second algorithm deals with the sce- 
nario in which the target distribution p is (e/4, t)-flat but no flat decomposition is provided to the 
learner. We show that in such a setting we can construct a {p,£,0{t/e))-Qat decomposition V of 
[n], and then we can simply use this V to run Learn-Known-Decomposition. 

The basic subroutine RiGHT-lNTERVAL will be useful here (and later). It takes as input an explicit 
description of a distribution q over [n], an interval J = [a, b] C [n], and a threshold r > 0. It returns 
the longest interval in [a, b] that ends at b and has mass at most r under q. If no such interval 
exists then q{b) must exceeds r, and the subroutine simply returns the singleton interval [b,b]. 



Subroutine RlGHT-lNTERVAL(g, J, r): 

Input: explicit description of distribution q; interval J = [a,b]; threshold r 

1. If q{b) > T then set i' = b, otherwise set i' = min{a < i < b \ q{[i, b]) < r}. 

2. Return [i',b]. 



The algorithm to construct a decomposition is given below: 

Algorithm Construct-Decomposition(p, t, e, 5): 

Input: sample access to unknown distribution p over [n]; parameter r; 
accuracy parameter e; confidence parameter 6 

1. Draw m = 0((l/r + log 1/5) /e^) samples to obtain an empirical distribution p, 

2. Set J= [7i],V = 0. 

3. While J / 0: 

(a) Let / be the interval returned by RlGHT-lNTERVAL(pm, J, r). 

(b) Add I toV and set J = J \ /. 

4. Return V. 



Theorem 3.2. Let <t be a class of [£/A,t)-flat distributions over [n]. Then for any p € Al- 
gorithm Construct-Decomposition(p, e/(4t),e,(5) draws 0{t/e^ + \og{l/ 5)/ e^) samples from p, 
and with probability at least 1 — 6 outputs a {p, e, 8t/£)-flat decomposition V of [n] . Its running time 
is 0((l/r + log(l/(5)) • log(n)/e^) bit operations. 

^Formally, define Ai — {[a, 6] | 1 < a < & < ?i} U {0} as the collection of all intervals over [n], including the empty 
interval. Then As = {h U ■ ■ ■ U L \ h, ■ ■ ■ , Is e Ai}. 
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To prove the above theorem we will need the following elementary fact about refinements: 



Lemma 3.1 ( |DDS+13[ Lemma 4]). Let p be any distribution over [n] and let X — be a 

{p,e,t)-flat decomposition of [n]. If J = {JiYi=i is a refinement of I, then J' is a {p,2€,t')-flat 
decomposition of [n] . 

We will also use the following simple observation about the Right-Interval subroutine: 

Observation 3.1. Suppose RiGHT-lNTERVAL(g, J, r) returns an interval I ^ J and Right- 
lNTERVAL(g, J\I,t) returns I' . Then q{I) + q{I') > r. 

Proof of Theorem \3.2l Let r =^ e/{4t). By Observation 13 -H the partition V that the algorithm 
constructs must contain at most 2/r intervals. Let Q be the common refinement of V and a 
(p, e/4, t)-flat decomposition of [n] (the existence of such a decomposition is guaranteed because 
every distribution in C is (e/4, t)-flat). Now note that 

Since Q is a refinement of the (p, e/4, t)-fiat decomposition of [n], Lemma l3 . II implies that the first 
term on the RHS is at most e/2. It remains to bound A = d^viP^^^^^^P^'^^^^^)- Fix any interval 
I & V and let us consider the contribution 

(l/2)^|/at(Q)(^.)_^fiat(P)(^.)| 

of / to A. If / € "P n Q then the contribution to A is zero; on the other hand, if / G "P \ Q then 
the contribution to A is at most p{I)/2. Thus the total contribution summed across all / G "P is 
at most (1/2) ^j^-p^QPil)- Now we observe that with probability at least 1 — 5 we have 

(1/2) E p(/)<e/4 + (l/2) E p^{I), (1) 
IeV\Q l€P\Q 

where the inequality follows from the fact that d^v (^p^^^^'P\ (Pm)^^*^^^) < £/4 by Proposition 13.11 
If / G P \ Q then I cannot be a singleton, and hence Pm{I) < by definition of RiGHT-lNTERVAL. 
Finally, it is easy to see that at most t intervals / in P do not belong to Q (because Q is the 
common refinement of V and a partition of [n] into at most t intervals). Thus the second term on 
RHS of Eq. ([T|) is at most tr = e/A. Hence A < e/2 and the theorem is proved. □ 

Our algorithm to learn an unknown (e/4, t)-flat distribution is now very simple: 

Algorithm Learn-Unknown-Decomposition(p, t, e, 5): 

Input: sample access to unknown distribution p over [n]; parameter t\ 
accuracy parameter e; confidence parameter 6 

1. Run Construct-Decomposition(p, e/(4t),e, 5/2) to obtain a (p, e, 8t/e)-fiat decompo- 
sition V of [n]. 

2. Run Learn-Known-Decomposition(p, P, e, (5/2) and return the hypothesis h that it 
outputs. 
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The following is now immediate: 



Theorem 3.3. Let C be a class of {e/4:,t)-flat distributions over [n]. Then for any p <t, Algo- 
rithm Learn-Unknown-Decomposition(p, 5) draws 0(t/e^ + log(l/5)/e^) samples from p, 
and with probability at least 1 — 5 outputs a hypothesis distribution h satisfying d"[v{p^h) < e. Its 
running time is 0(log(n) • (t/e^ + log(l/(5)/e^)) bit operations. 



3.3 Main result (third scenario): learning a mixture of flat distributions. We have 
arrived at the scenario of real interest to us, namely learning an unknown mixture of k distributions 
each of which has an (unknown) flat decomposition. The key to learning such distributions is the 
following structural result, which says that any such mixture must itself have a flat decomposition: 

Lemma 3.2. Let (t be a class of {e,t)-flat distributions over [n], and let p be any k-mixture of 
distributions in £. Then p is a {2e,kt)-fiat distribution. 

Proof. Let p = X]j=i l^jPj ^'^ ^ /c-mixture of components pi, . . . € C Let Vj denote the {pj, e, t)- 
flat decomposition of [n] corresponding to pj, and let V be the common refinement ofVi,V2,...,'Pk. 
It is clear that V contains at most kt intervals. By Lemma [3. "P is a (pj, 2e, kt)-fiat decomposition 
for every pj . Hence we have 



dTV (p,P 



< EHdTv(p„{Pjr'^''^) (2) 
i=i ^ ^ 

< 2e (3) 

where ([2]) is the triangle inequality and ([3|) follows from the fact that the expression in ([2]) is a 
nonnegative convex combination of terms bounded from above by 2e. □ 

Given Lemma 13.21 the desired mixture learning algorithm follows immediately from the results of 
the previous subsection: 

Corollary 3.1 (see Theorem II. ip . Let C be a class of {e/ 8, t)- fiat distributions over [n], and letp be 
any k-mixture of distributions in C. Then Algorithm Learn-Unknown-Decomposition(p, kt, e, 6) 
draws 0{kt/e^ + log(l/(5) /e^) samples from p, and with probability at least 1 — 5 outputs a hypothe- 
sis distribution h satisfying dTv{Pjh) ^ Its running time is 0(log(n) • {kt/e^ + log(l/(5)/e^)) bit 
operations. 



4 Learning mixtures of log-concave distributions 

In this section we apply our general method from Section [3] to learn log-concave distributions over 
[n] and mixtures of such distributions. We start with a formal definition: 

Definition 4.1. A probability distribution p over [n] is said to be log-concave if it satisfies the 
following conditions: (i) if 1 < i < j < k < n are such that p{i)p{k) > then p{j) > 0; and (ii) 
Pik)"^ > p{k - l)p{k + 1) for all k e [n]. 

We note that while some of the literature on discrete log-concave distributions states that the 
definition consists solely of item (ii) above, item (i) is in fact necessary as well since without it 
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log-concave distributions need not even be unimodal (see the discussion following Definition 2.3 of 
[FBRllj ). 

In Section l4.ll we give an efficient algorithm which constructs an (e, 0(log(l/e)/e))-flat decom- 
position of any target log-concave distribution. Combining this with Algorithm Learn-Known- 
Decomposition we obtain an 0(l/e'^)-sample algorithm for learning a single discrete log-concave 
distribution, and combining it with Corollarv 13.11 we obtain a k ■ 0(l/e'^)-sample algorithm for 
learning a A;- mixture of log-concave distributions. 



4.1 Constructing a fiat decomposition given samples from a log-concave distribution. 

We recall the well-known fact that log-concavity implies unimodality (see e.g. |KG71j ). Thus, it is 
useful to analyze log-concave distributions which additionally are monotone (since a general log- 
concave distribution can be viewed as consisting of two such pieces). With this motivation we give 
the following lemma: 

Lemma 4.1. Let p he a distribution over [n] that is non- decreasing and log-concave on [1,6] C [n]. 
Let L = [a, h] he an interval of mass p{L) = t, and suppose that the interval J = [1, a — 1] has mass 
p{J) = cj > 0. Then 

p{b)/p{a) < 1 + r/cj. 

dcf 

Proof. Let s = |I| = 6 — a + 1 be the length of /. We decompose J into intervals Ji, . . . ,Jt of 
length s, starting from the right. More precisely, 

Jj =^ / - js =[a- js, b - js] 

for 1 < J < t *== \{a — l)/s]. The leftmost interval Jt may contain non-positive integers; for this 
reason define p{i) '= for non-positive i (note that the new distribution is still log-concave). Also 
define Jo *== / = [a,b]. Let A '= p{b)/p{a). We claim that 

p{i-s)<{l/X)-p{i) (4) 

for 1 < i < 6. Eq. Q holds for i = b, since p{b — s) < p{a) by the non-decreasing property. 
The general case i < b follows by induction and using the fact that the ratio p{i — l)/p{i) is 
non-decreasing in i for any log-concave distribution (an immediate consequence of the definition of 
log-concavity) . 

It is easy to see that Eq. (jl]) implies 

p{J,+i)<{l/\)-p{Jj) 
for < j < t. Since the intervals have geometrically decreasing mass, this implies that 

i<j<t i>i A — i 

Rearranging yields the desired inequality. □ 



We will also use the following elementary fact: 

Fact 4.1. Let p he a distribution over [n] and L C [n] be an interval such that maxij^j p{i)/p{j) < 
1 + rj (i.e., p is rj-multiplicatively close to uniform over the interval I). Then the flattened sub- 
distribution p^^^'^^^i) =^p(/)/|/| satisfies dTviP^ ,P^'^^''^^) ^ V 'Pi^)- 
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We are now ready to present and analyze our algorithm Decompose-Log-Concave that draws 
samples from an unknown log-concave distribution and outputs a flat decomposition. The algorithm 
simply runs the general algorithm Construct-Decomposition with an appropriate choice of 
parameters. However the analysis will not go via the "generic" Theorem 13.21 (which would yield a 
weaker bound) but instead uses Lemma |4.H which is specific to log-concave distributions. 

Algorithm Decompose-Log-Concave(p, e, 5): 

Input: sample access to unknown log-concave distribution p over [n]; 
accuracy parameter e; confidence parameter 6 

1. Set r = e(e/log(l/e)). 

2. Run Construct-Decomposition(p, t, e, 5) and return the decomposition V that it 
yields. 



Our main theorem in this section is the following: 

Theorem 4.2. For any log-concave distribution p over [n], Algorithm Decompose-LogConcave(p, 
draws 0(log(l/e)/e^ -1- log(l/5)(log(l/e))^/e^) samples from p and with probability at least 1 — 6 
constructs a decomposition V that is {p,e,0{log{l/e)/e))-flat. 

Proof. We first note that the number of intervals in V is at most 2 /r by Observation 13.11 this will 
be useful below. We may also assume that dx{p,pm) < t, where pm is the empirical distribution 
obtained in Step 1 of Construct-Decomposition; this inequality holds with probability at least 
1 — d, as follows by a combined application of Theorems 12.11 and 12.21 Since p is log-concave, it is 
unimodal. Let be a mode of p. 

Let Vl = {I & V \ I C [1, io ~ 1]} be the collection of intervals to the left of io. We now bound the 

contribution of intervals in Vl to A '= dTV {p,p^^^^^^) ■ Let /i, . . . , /^^ be the intervals in Vl listed 
from left to right. Let Jj = Uji^jlji be the union of intervals to the left of Ij. If Ij is a singleton, 
its contribution to A is zero. Otherwise, 

p{Ij) <Pm{Ij)+T< It 

by the r-closeness of p and pm in Kolmogorov distance and the definition of Right-Interval. 
Also, by Observation 13. H Pm{Jj) > [{j — 1)/2Jt > ((j — l)/2 — l)r, and hence 

p{Jj)>Pm{Jj)-r>TU-5)/2, 
again by closeness in Kolmogorov distance. 
Since p is non-decreasing on [l,io ~ 1]) we have 

pi J _pflat(/,) 

for j > 5, by Lemma 14.11 and Fact 14.11 using the upper and lower bounds on p{Ij) and p{Jj) 
respectively. Consequently, _ j)flat(/j) < Qf^^jj^ j ^ [tL\- Summing this inequality, we 

get 

E ll/-/"*('^'l|i< E 0(r/j)=0(rlog(l/r)). 

3<tL j<tL 



< 
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The right-hand side above is at most e/2 by our choice of r (with an appropriate constant in the 
big-oh) . 

Similarly, let Vr = {I G "P | / C [io + 1,?^]} be the collection of intervals to the right of zq. 
An identical analysis (using the obvious analogue of Lemma 14.11 for non-increasing log-concave 
distributions on [iq + l,n]) shows that the contribution of intervals in Vr to A is at most e/2. 

Finally, let Iq V he the interval containing iq. If Iq is a singleton, it does not contribute to A. 
Otherwise, Pm{Io) ^ and p{Io) < 2r, hence the contribution of Iq to A is at most 2r. 

Combining all three cases. 



Our claimed upper bounds follow from the above theorem by using our framework of Section O 
Indeed, it is clear that we can learn any unknown log-concave distribution by running Algo- 
rithm Decompose-Log-Concave(p, e, 6/2) to obtain a decomposition V and then Algorithm Learn- 
Known-Decomposition(p, P, e, (5/2) to obtain a hypothesis distribution h: 

Corollary 4.1. Given sample access to a log-concave distribution p over [n], there is an algo- 
rithm Learn-Log-Concave(p, e, (5) that uses 0(log(l/5) log(l/e)/e^) samples from p and with 
probability at least 1 — 6 outputs a distribution h such that dTviPth) < e. Its running time is 
0(log(n) • {l/e^ -I- log(l/5)/e^)) bit operations. 

Theorem 14.21 of course implies that every log-concave distribution p is (e, 0(log(l/e)/e))-flat. We 
may thus apply Corollary 13.11 and obtain our main learning result for fc-mixtures of log-concave 
distributions: 

Corollary 4.2 (see Theorem II. 2p . Let p be any k-mixture of log-concave distributions over [n]. 
There is an algorithm Learn-Log-Concave-Mixture(p, /c, e, (5) that draws 0{k log{l/e)/e^ + 
log(l/(5)/e^) samples from p and with probability at least 1 — 6 outputs a distribution h such that 
dTviP:^) ^ running time is 0(log(n) • (A;log(l/e)/e'* -\-\og{l/ 6) / e^)) bit operations. 

Lower bounds. It is shown in [DLOU Lemma 15.1] that learning a continuous distribution whose 
density is bounded and convex over [0,1] to accuracy e requires il((l/e)^/^) samples. An easy 
adaptation of this argument implies the same result for a bounded concave density over [0, 1]. By 
an appropriate discretization procedure, one can show that learning a discrete concave density 
over [n] requires r2((l/e)^/^) samples for all e > l/n^^^\ Since a discrete concave distribution is 
also log-concave, the same lower bound holds for this case too. For the case of fe-mixtures, we 
may consider a uniform mixture of k component distributions where the i-th distribution in the 
mixture is supported on [1 -|- (i — l)n/k,in/k] and is log-concave on its support. It is clear that 
each component distribution is log-concave over [n], and it is not difficult to see that in order to 
learn such a mixture to accuracy e, at least 9/10 of the component distributions must be learned 
to total variation distance at most lOe. We thus get that for k < n^~^W ^nd e > l/nP^^\ any 
algorithm for learning a mixture of k log-concave distributions to accuracy e must use Cl{ke~^^'^) 



p-p 



,flat{P) < + e/2 + 2t< 2e. 




□ 



samples. 
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5 Learning mixtures of MHR distributions 



In this section we apply our general method from Section [3] to learn monotone hazard rate (MHR) 
distributions over [n] and mixtures of such distributions. 

Definition 5.1. Let p be a distribution supported in [n]. The hazard rate of p is the function 
H{i) =^ ^^~^~{')' ^j>iP^^^ ~ ^ ^f^^'^ ^6 say H{i) = +oo. We say that p has monotone hazard 
rate (MHR) if H[i) is a non- decreasing function over [n]. 

It is known that every log-concave distribution over [n] is MHR but the converse is not true, as can 
easily be seen from the fact that every monotone non-decreasing distribution over [n] is MHR. 

In Section l5.ll we prove that every MHR distribution over [n] has an (e, 0(log(n/e)/e))-flat de- 
composition. We combine this with our general results from Section [3] to get learning results for 
mixtures of MHR distributions. 



5.1 Learning a single MHR distribution. Our algorithm to construct a flat decomposition 
of an MHR distribution p is Decompose-MHR, given below. Note that this algorithm takes an 
explicit description of p as input and does not draw any samples from p. Roughly speaking, the 
algorithm works by partitioning [n] into intervals such that within each interval the value of p never 
deviates from its value at the leftmost point of the interval by a multiplicative factor of more than 
(l + e/8). 



Algorithm Decompose-MHR(p, e): 




Input: explicit description of MHR distribution p over [n]; accuracy parameter e > 




1. Set J = [n] and initialize Q to be the empty set. 




2. Let / be the interval returned by Right-Interval(p, J, e/8), and I' be the 
returned by Right-Interval(p, J \ /, e/8). Set J = J \ (/ U I'). 


interval 


3. Set i G J to be the smallest integer such that p{i) > ej (4n). If no such i exists, let /" = J 
and go to Step 5. Otherwise, let /" = [1, z — 1] and J = J \ I" . 


4. While J / 0: 




(a) Let j G J be the smallest integer such that either p{j) > (1 -|- e/8)p{i) or 
1-1-^/8^(0 holds. If no such j exists let = J, otherwise, let = [i,j — 1] 


v{j) < 


(b) Add to Q, and set J = J \ /"'. 




(c) Let i = j. 




5. Return P= QU {/,/',/"}. 





Our first lemma for the analysis of Decompose-MHR states that MHR distributions satisfy a 
condition that is similar to being monotone non-decreasing: 

Lemma 5.1. Let p be an MHR distribution over [n]. Let L = [a,b] C [n] be an interval, and 
R = [b + l,n] be the elements to the right of I. Let i] '= p{L)/p{R). Then p[b + 1) > jr-pia). 
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Proof. If p{b + 1) > p{a) then p{b + 1) > Y^p(a) holds directly, so for the rest of the proof we may 
assume that p{b + 1) < p{a). 

By the definition of the MHR condition we have , t ^["'} i \ < I t [^X^\\ and hence 

■' p{[a+l,n\) — p{[b+2,n\) 

P{[a,n]) ^ p{[b + l,n]) 
p{a) - p{b+l) 



Thus we obtain 



as desired. 



p{\b + 1, nl) , , 1 
P{b+l) > 1^ = T— " 



□ 

Let Q = {/i,/2, . . . , /|Q|}, with = [ai,bi], 1 < i < \Q\, where < Cj+i. Let Q' = {/j e Q : 
p{ai) > p(aj+i)} and Q" = {li € Q : p{ai) < p(aj_|_i)}. Thus, Q' consists of those intervals / in Q 
which are such that the following interval's initial value is significantly smaller than the initial value 
of I, and Q" consists of those I Q for which the following interval's initial value is significantly 
larger than the initial value of I. We also denote Ri = [ai+i,n]. For convenience, we also let 

"IQI+l = ^|S| + 1- 

We first bound the "total multiplicative decrease in p" across all intervals in Q': 
Lemma 5.2. We have Ur ao' 4^ < -■ 

iiJiey p(ai+-i) — e 

Proof. Observation 13.11 implies that the total probability mass p{I U /') on intervals / and /' is at 
least e/8. We thus have 

pjai) ^ p{I,)+p{R,) ^ p{h)+p{R,) _ p{h)+p{Ri) ^ 1 
/,GQ'P(a*+i) ~ /,eS' P{Ri) ~ h&Q Pi^i) P{R\Q\) ~ 

where the first inequality follows from Lemma [5. 11 the second inequality is self-evident, the equality 
follows from the telescoping product, and the final inequality is because p{I U /') > e/8. □ 

At this point we can bound the number of intervals produced by Decompose-MHR: 

Lemma 5.3. Step 4 of Algorithm Decompose-MHR adds at most 0(log(n/e)/e) intervals to Q. 

Proof. We first bound the number of intervals in Q' . Let the intervals in Q' be /[, • • • ^\q'\^ where 
I'j = [a'j,b'j] and a'l > a'2 > • • • > cl'^q/^- Observation 13.11 implies that the total probability mass 
p{I U /') is at least e/8. Hence, p{[bi + 1, n]) is at least e/8 and we have p{Ri) > e/8. For j > 1 it 
holds 

p(i2;.)>(e/8)(l + e/8r^ (5) 
Consequently the number of intervals in Q' is bounded by 0(log(l/e)/e). 
Now we bound the number of intervals in Q". We consider the value of H/ieQ EiSn+ll- 
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Since p{a\Q\+i) < 1 and p{ai) > e/(4?i), the above is at most 4n/e; using Lemma [5^ we get that 

n < (An/e) ■ {8/e) = {32n/e'). 

On the other hand, for every /j G Q" we have that ^^"^"^^^^ > (1 + e/8). Consequently there can be 
at most 0((l/e) log(n/e)) intervals in Q" , and the proof is complete. □ 

It remains only to show that V is actually a flat decomposition of p: 

Theorem 5.1. Algorithm Decompose-MHR outputs apartitionV of[n] that is {p,e,log{n/e)/e))- 
flat. 

Proof. Lemma 15.31 shows that V contains at most 0(log(n/e)/e) intervals, so it suffices to argue 
that dTV (p,/''*^^)) < e. 

We first consider the two rightmost intervals / and /'. If |/| = 1 then clearly dxy {p^ ,p^^^^^^) = 0, 
and if \I\ > 1 then p(J) < e/8 and consequently dxi/ {p^,p^^^^^^) < e/8. Identical reasoning applies 
to /'. For the leftmost interval /", we have that p{I") < e/4, so d^vip^" ,p^^^^^"^) < e/4. Thus, so 
far we have shown that the contribution to d^v {p,p^^^^^^) from / U /' U /" is at most e/2. 

Now for each interval in Q, we have 

max 44 < (1 + = 1 + e/4 + 

Since the total probability mass on intervals / and /' is at least e/8 by Observation 13.11 the total 
probability mass on intervals in Q is at most 1 — e/8. An easy calculation using Observation 14.11 
shows that the total contribution to d^^yip^P^'^^^'^^) from intervals in Q is at most e/4, and the 
theorem is proved. □ 

Applying Corollary 13.11 we get our main learning result for mixtures of MHR distributions: 

Corollary 5.1 (see Theorem 1 1.3|) . Letp he any k-mixture of MHR distributions over [n]. There is 
an algorithm Learn-MHR-Mixture(p, A:, e, 5) that draws 0(A;log(n/e)/e^ +log(l/()')/e^) samples 
from p and with probability at least 1 — 6 outputs a distribution h such that d-^v{p,h) < e. Its 
running time is 0((logn)^ • (A;log(l/e)/e^ + log(l/5)/e^)) bit operations. 



Lower bounds. By adapting a lower bound of |Bir87aj (for monotone distributions over a con- 
tinuous interval) it can be shown that for e > l/n^^^\ any algorithm for learning a monotone 
distribution over [n] to accuracy e must use 0(log(n)/e^) samples. We may consider a uniform 
mixture of k component distributions where the i-th distribution in the mixture is supported on 
and monotone non-decreasing over [l + (i — l)n/A;, in/k]. Each component distribution is MHR (over 
the entire domain). The same argument as in the log-concave case implies that, for k < n^~^W 
and e > l/n^^^\ any algorithm for learning a mixture of k MHR distributions to accuracy e must 
use Q{klog{n)/e^) samples. 
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6 Learning mixtures of unimodal and t-modal distributions 

In this section we apply our general method from Section [3] to learn mixtures of unimodal (and, 
more generally, t-modal) distributions over [n]. Here our task is quite easy because of a result of L. 
Birge |Bir87bj which essentially provides us with the desired flat decompositions. H 

We begin by defining unimodal and t-modal distributions over [77,]: 

Definition 6.1. A distribution p over [n] is unimodal if there exists i € [n] such that p is non- 
decreasing over [l,i] and non-increasing over [i,n]. For t > 1, distribution p over [n] is t-modal if 
there is a partition of [n] into t intervals /i, . . . ,/< such that the sub -distributions p^^ , . . . ,p^^ are 
unimodal. 

By adapting a construction of Birge (proved in |Bir87bj for distributions over the continuous real 
line) to the discrete domain [n], |DDS"'"13] established the following: 

Theorem 6.1 r |DDS+13[ Theorem 5]). Let p be any monotone distribution ( either non-increasing 
or non- decreasing) over [n]. Then p is {e,0{log{n)/e))-flat. 

We note that it can be shown (using the same construction that is used in the 0(log(n)/e'^) sample 
complexity lower bound of |Bir87a| for learning monotone distributions) that 0(log(n)/e) is the 
best possible bound for the number of intervals required in Theoorem 16.11 

An immediate consequence of Theorem lO.ll is that any unimodal distribution over [n] is (e, 0(log(n)/e))- 
flat, and any t-modal distribution over [n] is (e,0(i • log(n)/e))-flat. Using Corollary 13.11 we thus 
obtain the following results for learning mixtures of unimodal or t-modal distributions: 

Corollary 6.1 (see Theorem II. 4p . For any t > 1, let p be any k-mixture of t-modal distri- 
butions over [n]. There is an algorithm Learn-Multi-modal-Mixture(p, A;, t, e, (5) that draws 
0{kt\og{n) / e'^ + \og{l / 5) / e^) samples from p and with probability at least 1 — 5 outputs a distri- 
bution h such that d^vip^h) < e. Its running time is 0(log(n) • {kt\og{n)/£^ + log(l/5)/e^)) bit 
operations. 

Lower bounds. The lower bound arguments we gave for mixtures of MHR distributions (which are 
based on Birge 's lower bounds for learning monotone distributions) apply unchanged for mixtures of 
unimodal distributions, since every distribution which is supported on and monotone non-decreasing 
over [1 + (i — l)n/k.,in/k] is unimodal over [n\. 

7 Conclusions and future work 

This work introduces a simple general approach to learning mixtures of "structured" distributions 
over discrete domains. We illustrate the usefulness of our approach by showing it yields nearly 
optimal algorithms for learning mixtures of natural and well-studied classes (log-concave, MHR 
and unimodal) and in the process we establish novel structural properties of these classes. 

Are there any other natural distribution classes for which our general framework is applicable? 
We suspect so. At the technical level, the linear dependence on the parameters k and t in the 
sample complexity of Theorem 11.11 is optimal (up to constant factors). It would be interesting to 
improve the dependence on 1 /e from cubic down to quadratic (which would be best possible) with 
an efficient algorithm. 

■^We note that Birge's structural result was obtained as part of an efficient learning algorithm for monotone 
distributions; Birge subsequently gave an efficient learning algorithm for unimodal distributions [Bir97] . However, 
we are not aware of work prior to ours on learning mixtures of unimodal or f-modal distributions. 
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A Proof of Proposition 11.11 

At a high level, the algorithm A' works by drawing a large set of samples from the target mixture and 
trying all possible ways of partitioning the sample into k disjoint subsamples. For each partition 
of the sample it runs algorithm A over each subsample and combines the resulting hypothesis 
distributions (guessing the mixture weights) to obtain a hypothesis mixture distribution. Finally, 
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a "hypothesis testing" procedure is used to identify a high-accuracy hypothesis from the collection 
of all hypotheses distributions obtained in this way. 

More precisely, let p denote the unknown target /c-mixture of distributions from <t. Algorithm A' 
works as follows: 

1. Draw a sample S oi M = 0{k/e) ■ m{n, e/20) ■ log{5k/5) samples from p. 

2. For each possible way of partitioning S into k disjoint subsamples S = {Si, . . . , S^) such 
that each \Si\ > m{n, e/20) ■ log{5k/6), run algorithm A a total of k times, using Si as 
the input sample for the i-th run, to obtain hypothesis distributions hf,...,hf. For each 
vector /X = (;Ui, . . . , /x^) of non- negative mixing weights that sum to 1 and satisfy fii = 
(integer) •e/(20A;), let be the mixture distribution J2i=if^i^i- 

3. Draw M' = 0{M\ogk + klog{k/e)) ■ log(5/(5)/e^ samples from p and use them to run the 
"hypothesis testing" routine described in Lemma 11 of |DDS12b] over all hypotheses 
obtained in the previous step. Output the hypothesis distribution that this routine outputs. 

We now proceed with the analysis of the algorithm. Let p = Yli=i '^iPi denote the target k- 
mixture, where the mixing weights and pi, ■ ■ ■ ,Pk are the components. Without loss 

of generality we may assume that i = 1, . . . ,i are the components such that the mixing weights 
Ki, . . . ,Ki are at least e/(20/c). A standard "balls in bins" analysis (see |NS60j ) implies that with 
probability at least 1 — 5/5 the sample S contains at least m{n, e/20) ■ log{5k/6) draws from each 
component pi,...,pi; we assume going forth that this is indeed the case. Thus there will be 
some partition S = {Si, . . . ,5^) which is such that each Si with 1 < i < i consists entirely of 
samples drawn from the component pi. For this S, we have that with failure probability at most 
{6/{5k)) ■ k < 6/5, each hypothesis distribution hf for 1 < i < £ satisfies dTv{Pi:hf) < e/20. 
Now let fi* denote the vector of hypothesis mixing weights (as described in Step 2) that has 
llJ'i — Ki\ < e/{20k) for all i = 1, . . . , /c. It is not difficult to show that the hypothesis mixture 
distribution h* = h^* satisfies dT^v{h*,p) < 3e/20 < e/6, where e/20 comes from the errors 
d^v{Pi7 hf) for i < £, e/20 comes from the inaccuracy in the mixing weights, and e/20 comes from 
the (at most k) components pj with j > i that each have mixing weight at most e/{20k). 

Thus we have established that there is at least one hypothesis distribution h* among the /ij^'s that 
has di^y{p,h*) < e/6. There are at most = /e*^ • {20k/e)^ hypotheses generated in Step 2, 
so the algorithm of Lemma 11 of [DDS12b] requires 0(log A^) log (5/5) /e^ < M' samples, and with 
probability at least 1 — 6/5 it outputs a hypothesis distribution h that has d'Yy{p,h) < e. The 
overall probability of outputting an e-accurate hypothesis is at least 1 — 6, and the proposition is 
proved. 
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